#BPE tokenizer

Meet Gigatoken: A Rust BPE Tokenizer that Encodes Text at 24.53 GB/s, up to 989x Faster than HuggingFace Tokenizers

Gigatoken, a new Rust-based BPE tokenizer, achieves text encoding speeds of up to 24.53 GB/s, outperforming HuggingFace tokenizers by 989x and tiktoken by 681x.